Approach

This analysis estimates the impact of Balto usage on Outbound transfer rate for SelectQuote agents. To estimate this impact, the following process is used:

Estimate a statistical (mixed logistic) model that is a function of who the agent is, whether the agent is a Balto user, time, their skill level, and vendor campaign. We additionally interact the indicator for being a Balto user with agent skill level, allowing the effect of Balto to vary across skill level.
Calculate a 95% confidence interval around the estimated Balto effect for each skill level, then translate the estimates to a percentage deviation from the pre-test average for each respective skill.
Visualize the estimated effects.

We employ this approach to account for all potential confounding factors linked to the KPI of interest that could bias results in the presence of insufficient group randomization. Taking this approach also avoids unnecessary aggregations and maximizes available information to make statistical inference.

We find a positive statistically significant effect for Dual Skilled agents, and a statistically insignificant negative effect for Specialty agents.

Results

Summary Tables

Dual Skilled Agents

Specialty Agents

Visualizations

Technical Details

Model Output

	Transfer Rate
Predictors	Odds Ratios	95% CI
Intercept	0.01 ^***	0.00 – 0.01
Balto User = Yes	2.57 ^*	0.94 – 6.99
Agent Level = Specialty	3.35 ^**	1.05 – 10.75
Balto User = Yes * Agent Level = Specialty	0.39	0.11 – 1.34
Linear Time Trend	0.76 ^***	0.64 – 0.91
Random Effects
σ²	3.29
τ₀₀ _Agent	0.00
τ₀₀ _{VendorCampaignName}	0.00
N _{VendorCampaignName}	18
N _Agent	28
AIC	1485.773
log-Likelihood	-735.887
p<0.1 p<0.05 * p<0.01

Marginal Effects

Data

Balto.ai

SelectQuote Pilot Analysis

Outbound transfer rate

Balto Data Science - 20 October 2021